Euroscene 2/Euroscene 2.iso/USEFUL/libs/FileSys/XFH/XFH.doc

Text File | 1995-11-01 | 20KB | 390 lines

                  Xpk File System Handler v1.12

                      By Kristian Nielsen


This package may be distributed freely for non-commercial purposes along
with Xpk or on its own, provided that all files and subdirectories are
left intact. However, all rights remain with the author. And as usual,
the author will in no way be held responsible for any damage, directly,
implicit or otherwise, resulting from the use of this program. After
all, you get this program for free; use it or not as you like, and at
your own risk.

The file 'README' (which should be distributed alone with this file) gives
information on how to get started quickly while this text gives a more
technical discussion of the XFH file system handler.



                              --- O ---


What's possible.

   XFH, when used with Xpk, makes it possible to store data in compressed
format without this being visible to the user or to application programs -
XFH will make the compressed data appear like ordinary files. This is by
no means a new idea. In MS-DOS world, a (commercial) program called
'Stacker' has been available for some time, which makes it possible to
compress a whole partition on a harddisk. On the amiga, people have long
used powerpacker along with programs like PPMore to store data files in
packed formats (but relying on the application to recognise that the file
is in compressed format), and programs like PPPatch have been used to
change dos.library to recognise the powerpacker format automatically.
However, none of these approaches are perfect in all situations, and so
there is room for another alternative - the XFH.

   XFH works in conjuction with the Xpk approach to data compression. This
in itself gives a number of advantages - a flexible interface to the
compressing algorithms, lots of different packers available with the
possibility to add new ones etc. But compared to approaches like
'Stacker', there is an additional advantage: access to the compressed
data is not limited to XFH - most of the time, the XFH way will be the
most convenient way to access data, but if needed, the complete range of
Xpk applications is available to the user. The file orientated nature of
Xpk also means that XFH - like PPPatch etc., but unlike Stacker - will
co-exists nicely with any other Amiga filing system without the need to
set up any new partitions or prepare the disk with a special program. In
fact, if you happened to have a CD-Rom (read only) stuffed with
powerpacked files (or any other format that the current version of Xpk
supports), you could dump an XFH unit on top of it and largely forget
about the disk being compressed from then on.

   So, what is possible is to mount a XFH unit on top of each of one or
more standard AmigaDOS directories. It is now possible for programs to
access the directories as usual, data being decompressed and optionally
compressed as needed in a completely transparent way - programs will
never know the difference between compressed and uncompressed files -
while still having access to the compressed data using conventional
compress/uncompress programs. So the dream is that of doubling the
capacity of your hard- or floppydisks for zero cost.

   As an example of the possibilities of the XFH, I have been using it for
a couple of months for holding various doc files, metafont sources and
little used emacs scripts and shell commands on HD, saving in the order
of perhaps 60% file space with no noticeable degrading of overall system
performance.


                              --- O ---


What is the catch?

   Of course, as everyone knows, nothing comes for free. Obviously,
there is a speed penalty to compressing or uncompressing a file.  This
speed penalty will be highly dependent on the actual compression
algorithm and data media used; for example an algebraic compression
scheme used with a superfast HD will probably be rather slow, while a
fast algorithm may actually result in a speed-up of floppy access since
the disk access time saved by the smaller file size more that accounts
for the time taken decompressing. And of course with the coming of ever
more powerful CPU's the speed will be less of a problem. Some things
will always be slow, though. Especially directory listning is a
problem, since every single file has to be opened in order to check
whether the file is compressed or not. In fact, I'll admit that running
XFH from floppy on an unaccelerated amiga will sometimes seem a bit
slow. However, XFH was not written to be as fast and small as possible,
but rather to be flexible and expandable. And I'm sure a lot of users
will find it very useful even on 'small' amigas. On an A3000, XFH runs
like a dream, of course. And there is still the possibility for speed
improvements in later versions.

   Aside from the problem of speed, V1.12 of the XFH comes with a few
other limitations that I'm hoping to remove in later versions. Most
important is the lack of most of the new 2.0 packets, a lack that will
become more severe as more 2.0-only programs start to depend on these
packets. Another problem is that XFH in the present version is somewhat
memory-hungry, in that it will keep any compressed file completely
unpacked in memory as long as that file is open. This is mostly due to
limitations in the current Xpk interface, and I'm hoping to remove this
in a later release. However, currently this means that it is impossible
to open a compressed file if it is larger than available memory.

   A number of other problems to look out for are detailed in a later
section in this doc file. Most of these are things that could be fixed in
later versions.



                              --- O ---


Instructions for use.

   XFH is implemented as an AmigaDOS device handler. The 'L:' directory
contains examples of other such device handlers, and this is usually also
the best place to put XFH (the file 'XFH-Handler', to be precise). Like
other handlers, XFH must be mounted before it can be used. This can be
done by creating an entry for it in the 'DEVS:Mountlist' file (just the
mountlist for short) and issuing the command 'Mount <device>', where
<device> is the name given to the device in the mountlist. The supplied
file 'Devs/Mountlist' contains four example entries for the XFH; it might
be convenient to append this file to the end of 'DEVS:Mountlist'. The
most important part of each mountlist entry is the unit number (the
number used in the 'Unit = ...' line), since this is used to control what
files that particulary XFH unit should (de)compress, how it should do it
etc. Further information on the mechanism of mountlists and handlers in
general can be found in various places; I will not attempt any lenghty
explanation here.

   Unlike most other 'normal' device handlers, XFH needs additional
information to function correctly. Most importantly it needs to be told
what directory it should use as its root directory (that is, where it
should look for compressed files). In the simplest case, this can be done
by assigning 'XFH<n>:' to this directory, where <n> is the unit number as
given in the mountlist. However, XFH also provides 'options' which
provide much more flexibility and which MUST be used if the handler is to
compress files automatically as they are written by applications.

   In the current version all options must be set in files (hopefully a
later version will also provide an AREXX interface).  An option is
specified on a line of its own, and follows the same conventions as the
TOOLTYPES of the Workbench.  XFH has the concept of primary and
secondary option files.  The primary option file, for XFH unit number
<n>, is placed in the file 'S:Xfh/.xfhrc_<n>'.  The secondary option
file is named '.xfhrc' and is placed in the root directory of the XFH
unit (uncompressed!).  The settings in the secondary option file
overrides any settings in the primary option file, but there are some
restrictions on the options that can be used in the secondary option
file, see below.  Both files are optional, the handler will pick default
values for any options that are not set by the user.  The supplied
directory 's/xfh/' contains example primary option files for units 1-3.

   Boolean options can be specified with "NO" / "OFF" or. "YES" / "ON".
String options are specified by simply putting the string after the
'OPTION=' bit, no quotes are needed.

The available options are detailed below:

   ROOTDIR
   VOLUMENAME
   AUTOCOMPRESS
   PACKMODE
   STEPDOWN
   PASSWORD
   XPKPRIORITY
   TRUNCATEONPACK

Option ROOTDIR:
   This option is used to set the name of the directory that XFH: is to
reside in. For example 'ROOTDIR=Work:xfh' would cause XFH: to use that
directory as root. Note that this option can only be used in the primary
option file 's:xfh/.xfhrc_<unit>'. When used elsewhere it will simply be
ignored. If this option is not specified, XFH will use the assign
'XFH<n>:', where <n> is the unit number. This is mainly for compatibility
with earlier versions of XFH: ; using the ROOTDIR option is a much cleaner
way of doing this.

Option VOLUMENAME:
   This option is used to set the name of the XFH: volume. This is the name
that will be used in absolute path specifications, as well as the one
returned by Info(). Ie. if you go 'VOLUMENAME=Manuals', access can be by
'Manuals:' as well as by 'XHn:' (or whatever). Like ROOTDIR, this option can
only be used in the primary option file. If this option is not specified,
the default is to use the name of the directory in the underlying file
system, or (if this is itself the root of a volume) to use the name with
'XFH_' prepended. Again, this is mainly for compatibility with early
versions of XFH:.
   Note that this option can be seen as a replacement for the 'Relabel'
command (which isn't supported).

Option AUTOCOMPRESS:
   This options tells whether XFH: should attempt to compress the files
written through it. When this option is set, everytime a file written to
the XFH: is closed, an attempt will be made to compress the file to a
temporary file using Xpk. If this is succesfull, the temporary file will
be renamed to the original name and the uncompressed file will be deleted.
If the compression fails for any reason, the uncompressed file will simply
remain intact. This also means that if anything should go wrong during
compression (like a disk full error), it is unlikely that any data will
be lost since at least one of the two files should be intact (though
possibly with a wierd name). This option is OFF by default.

Option PACKMODE:
   This option selects the mode that Xpk should use when compressing files.
It is specified in the usual way when using Xpk. For example, to use 12-bit
BLZW compression, 'PACKMODE=BLZW.12' would be used. The default is to use
the NUKE compression ('PACKMODE=NUKE'). Of course, to use a specific
compression method, the nessesary sublibrary must be available in LIBS:.

Option STEPDOWN:
   This option controls the Xpk flag 'XPK_StepDown' during packing. If set,
it means that Xpk is allowed to reduce packing efficiency if nessesary to
save memory. Refer to the Xpk documentation for details. Default is OFF.

Option PASSWORD:
   This option is used to set the password that XFH: should pass on to
Xpk when compressing or uncompressing. It should be noted that there is
no attempt to keep this password safe from 'memory peekers'. Of course,
storing the password in the option file isn't actually safe either. The
latter problem will be solved when XFH gets an AREXX interface; until
then, a script something like this

   copy xfhoptions/.xfhrc xfhdir:         ; xfhdir: is the root of xfh:
   echo >> xfhdir:.xfhrc "PASSWORD=..."
   mount XFH:
   info >nil: xfh:                        ; Make sure it's really loaded.
   copy xfhoptions/.xfhrc xfhdir:
   echo >> xfhdir:.xfhrc "PASSWORD=***"   ; Really erase from disk.

could be used, though it still isn't completely safe. Finally note that
if AUTOCOMPRESS is requested, files may still be saved unencrypted in
low-memory or low-diskspace situations, and (depending on the underlying
file system / disk device) part of the unencrypted data may still be
physically stored on the disk after the deletion of the unencrypted file.
   If an attempt to open a file is made when XFH has been given the wrong
password, the open will fail with error code 212 (Object wrong type).

Option XPKPRIORITY:
   This option is used to set the task priority that should be used when
doing Xpk operations (compress/uncompress etc). Setting this to zero or
less will prevent XFH from stealing all CPU-time from tasks running at a
'normal' priority. Note that it is possible to set the priority of the
handler itself in the mountlist. If this option is not used, the handler
will keep whatever priority it is running ta when calling the Xpk library.

Option TRUNCATEONPACK:
   This option is somewhat technical in nature and can be safely ignored.
It is only used in case of an error occuring during the compression of a
file. In this case, the compressed file has to be deleted, and if this
option is set, the handler will try to call SetFileSize() first to truncate
the file to 0 bytes (perhaps saving the flushing of a few buffers).
However, due to sparse documentation I'm uncertain whether this feature is
stable, and hence it is OFF by default. Again, unless you are curious and
don't mind risking crashes/data losses, forget about this option.


In case of an error during the scanning of the option files the handler
will fail its initialisation and hence refuse to load with error code 114
(Bad Template). Sorry, but there are currently no real error messages
implented (this will hopefully be fixed in a later version). If the handler
refuses to work for no apparent reason, be sure to tripple-check your
option files for errors.




                              --- O ---


Limitations and known bugs.

   The error detection code in the initialisation part of the handler is
somewhat flaky - I've tried to make it resonably safe, but documentation
on the right way to start a handler is hard to find. What it means is
that it is a good idea to make sure that the handler is placed in L: and
that the nessesary Xpk libraries are placed in libs: before starting the
handler (remember, XFH won't be able to tell you the reason if it was
unable to initialise for some reason). Another subtle problem is that due
to a quirk of the device list locking, it is vital (using XFH v1.12) that
the handler for the directory that XFH is to sit in is already loaded.
Usually this will not be a problem; however, if you are using 3rd party
filesystem handlers that are mounted after boot-up, you can avoid
problems by accessing them before mounting XFH (for example by creating
an assign (not late-binding) to them).

   It should be noted that the way the XFH makes the same files available
by two different routes is not without its problems. One problem is
connected to the volume name - XFH tries to be smart about it, but it will
sometimes create a duplicate volume name which is a bad idea. To solve
this problem the VOLUMENAME option should be used in an option file.
Another problem when mounting XFH in the root dir appears when using the
'Leave Out' feature of the 2.0 Workbench. Here, the '.backdrop' is
duplicated in both volumes, making the left-out icons appear twice. I'm
working on a decent solution to this problem. Meanwhile, I would recommend
that XFH is mounted only on top of subdirectories.

   For convenience, the handler allows the opening of and writing to new
files. Also, XFH now facilitates automatic compression, as discussed in
the description of the AUTOCOMPRESS option. However, note that the
handler does not yet support opening existing files for writing (this is
technically refered to as MODE_READWRITE or ACTION_FINDUPDATE), and
writing to compressed files opened in shared mode (MODE_OLDFILE) also
does not work. This means that for example it is not possible to append
something to a file using the shell '>>' redirection. Still, keeping
this in mind it should nevertheless be possible to use the handler even
in directories where files are often being written to.

   Needless to say (but I'll say it anyway), you should not assign libs:
to a XFH unit unless you are absolutely sure of what you are doing. A
nice trap is to have the XFH call (and wait for) Xpk, which will then wait
for XFH to load a particular library for it... This also applies to s:
(since the option files are stored here).

   It should be noted that any given unit of the XFH binds to a directory,
not to a DOS device. This means that, currently, it is not possible to
have a XFH unit working like DFx: - any access will refer to the disk that
was in the drive when the handler was started, not to the disk currently
in the drive.

   The figures reported by the shell 'Info' command are somewhat strange.
The problem is that it isn't really possible to give sensible figures for
'NumBlocks' and 'NumBlocksUsed'. Currently, their values are the same as
those for the underlying file system.

   XFH has problems with exclusive locks (for example trying to obtain an
exclusive lock on "/" or "foo//" will always fail).

   Some people have experienced problems when using XFH with the
arp.library. This is because a bug/feature in the apr.library function
CompareLock() (it declares two locks equal only if the lock->fl_Key
fields are equal, which is illegal according to the 2.0 DOS manual).
Under 2.0, the program in the directory 'patcharp' can be used to patch
arp to use the correct 2.0 SameLock() call. Also, I know of a program
that remaps the arp calls to the corresponding 2.0 dos.library calls
(though I haven't tried it), it might help too.

   (This one is a bit technical in nature:) Currently, for reasons having
to do with the reading of the primary option file, 'S:' must to refer to
an assign (or a volume), not to a device like df0: or ram:. However,
since it is almost always an assign (to SYS:s or something), this
shouldn't be a big problem.

   It has been reported to me that some disk toolkit programs (like DiskX)
have problems with the "dummy.devide" entry in the mountlist. If you
experience any problems with this, you should try changing this entry to
something else (like "trackdisk.device" or "scsi.device") or omit it
entirely; it is not used by XFH and is only there for the sake of the
'Mount' command.


                              --- O ---


Theory of operation.

   XFH works by installing itself in the system as a file system handler
like DF0: or RAM:. However, unlike most file system handlers, which sit on
top of a device (or rest in themselves like RAM:), XFH sits on top of an
underlying file system handler (abbrevated to UFS) containing a mixture
of normal and compressed files. After initialising itself, XFH sits in a
loop receiving packets from AmigaDOS and other applications. Each packet
is examined and the appropriate action taken. For example, an open request
will cause XFH to open the given file, check whether it is compressed, and
if so unpack it to memory for later read requests.

   XFH is currently single-threaded, unlike the Commodore file systems
(this means that it is not possible for the XFH to service other requests
while waiting for the UFS). I'm hoping to fix this in some later version.



                              --- O ---


Acknowledgments.

   The author wishes to thank all the people that have participated in
the development of Xpk without which the XFH would not have been the
same. I am especially grateful to Urban D. Müller for helping me start
the whole concept of the XFH back in the summer of 1991 - without his
help the XFH is not likely to have been realised. Thanks also to the many
beta testers who helped me iron out as many bugs as possible before
release; your help work has been very valuable to me. And thanks must
go, of course, to the guys at Commodore for bringing to us the wonderful
Amiga.